On Batch-Scheduling Dags for Internet-Based Computing

The process of scheduling computations for Internet-based computing presents challenges not encountered with more traditional platforms for parallel and distributed computing. The looser coupling among participating computers makes it harder to utilize remote clients well and also raise the specter of a kind of “gridlock” that ensues when a computation stalls because no new tasks are eligible for execution. This paper studies the problem of scheduling computation-dags in a manner that renders tasks eligible for allocation to remote clients (hence for execution) at the maximum possible rate. Earlier work has developed a framework for studying this problem when a new task is allocated to a remote client as soon as it returns the results from an earlier-allocated task. The proof in that work that many dags cannot be scheduled optimally within this scheduling paradigm demonstrated the need for a companion scheduling theory that addresses the scheduling problem for all computation-dags. A new, batched, scheduling paradigm for InternetComputing is developed in this work. Although optimal batched schedules exist for every computation-dag, computing (successive steps of) such a schedule is shown to be NP-Hard, even for bipartite dags. In response, a polynomial-time algorithm is developed for producing optimal batched schedules for a rich family of dags that are constructed by “composing” tree-structured building-block dags. Finally, a fast heuristic schedule is developed for a class of “expansive” dags